DOCUMENT RESUME 



ED 473 390 



IR 058 620 



AUTHOR 

TITLE 

■ PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Pick, James B.; Kim, Jeff 

Program Assessment in an Undergraduate Information Systems 
Program: Prospects for Curricular and Programmatic 
Enhancement . 

2000 - 00-00 

15p.; In: Proceedings of the International Academy for 
Information Management Annual Conference (15th, Brisbane, 
Australia, December 6-10, 2000); see IR 058 611. 

Reports - Research (143) -- Speeches/Meeting Papers (150) 

EDRS Price MF01/PC01 Plus Postage. 

Curriculum Development; * Educational Assessment; ^Evaluation 
Methods; Higher Education; ^Information Science Education; 
^Information Systems; Models; Program Effectiveness; ^Program 
Evaluation; Undergraduate Study 
^University of Redlands CA 



ABSTRACT 

This paper offers an assessment model approach for program 
assessment of Information Systems (IS) undergraduate programs. The paper 
presents the case example of results of program assessment for the B.S. in 
Information Systems at University of Redlands (UR) . It demonstrates the 
usefulness of the approach to understanding the overall success of an IS 
degree program as well as details of program elements. The example points to 
many ways that the outcomes of assessment can be utilized for IS program 
improvement. Four tables include: knowledge attainment goals of BSIS major 
program; curriculum of bachelor of science in information systems (BSIS); 
correlations of important knowledge/competency areas for current BSIS; and 
correlations of career impact on alumni for important BSIS curricular areas. 
Also includes two figures. (Contains 18 references.) (AEF) 




Reproductions supplied by EDRS are the best that can be made 
from the original document. 



IR058620 



PROGRAM ASSESSMENT 
§ IN AN UNDERGRADUATE 

S INFORMATION SYSTEMS PROGRAM: 

g PROSPECTS FOR CURRICULAR AND 

PROGRAMMATIC ENHANCEMENT 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



-T-i— Case 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



Janies B. Pick 
University of Redlands 



Jeff Kim 

University of Redlands 



^U^fDEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

*2 This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



ABSTRACT 



This paper examines the assessment of an undergraduate program in Information Systems . It presents an 
approach to program assessment that has proven beneficial to one institution. The data from program 
assessment can be useful to stakeholders and forms a baseline for program improvement. 



INTRODUCTION 

A major trend in U.S. higher education of the last ten 
years has been towards educational assessment. 
Universities, accrediting organizations, professional 
academic associations, and government review panels 
have requested and often demanded assessment 
information from schools, departments, programs, 
projects, and individual faculty. One example is the 
AACSB accreditation standards for business schools, 
which require for accreditation that a variety of 
assessment processes be in place. In the field of 
information systems, not much has been reported in 
the literature on program assessment, yet it is 
important because IS programs are growing in size and 
their success needs to be monitored and informed steps 
taken to improve the curricula. 

This paper offers an assessment model approach for 
program assessment of IS undergraduate programs. It 



presents the case example of results of program 
assessment for the B.S. in Information Systems at 
University of Redlands (UR). It demonstrates the 
usefulness of the approach to understanding the overall 
success of an IS degree program as well as details of 
program elements. The example points to many ways 
that the outcomes of assessment can be utilized for IS 
program improvement. 

The B.S. in IS at UR is a 14 year old program that had 
undergone three major curricula upgrades, with a 
fourth one planned in the fall of 2000. The prior 
curricula changes were effected in the years 1986, 
1992, and 1997. The curricula were based on national 
model curricula, including the DPMA 90 curriculum, 
and the IS 97 curriculum (it was available in 
preliminary form two years prior to its final release in 
1997.) The Redlands B.S. in IS is a degree completion 
program that includes only junior and senior 
undergraduate years, with the first two years of work 
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done at another university or college. The Redlands 97 
curriculum’s knowledge attainment goals are shown in 
Table 1. The Redlands 97 IS curriculum consists of the 
IS major consisting of nine IS courses, as well as one 
humanities course, one upper division mathematics 
course, and four business courses (See Table 2). The 
nine IS courses encompass nearly all of the IS '97 
model curriculum, with the exception of several 
introductory courses. However, since the students 
enter as working adults, many with IS industry 

TABLE 1 

KNOWLEDGE ATTAINMENT GOALS OF BSIS 
MAJOR PROGRAM 

• Understand an apply structured programming 

techniques 

• Be conversant in fundamental C language 

capabilities 

• Understand the Graphical User Interface (GUI) 
principles 

• Be conversant in fundamental Visual Basic 

language capabilities 

• Understand the basic terms and components of the 
relational database model 

• Understand and be able to apply database 

application development using Access in 

conjunction with the VisualBasic programming 
language 

• Understand and apply the fundamental concepts 
and terminology associated with networking and 
telecommunications 

• Understand and be able to build internet and web- 
based applications 

• To appreciate the design and structure of 

geographic information systems as a 

decision_making tool 

• To demonstrate knowledge of decision support 
systems, executive information systems, and 
expert systems including some hands-on 
applications 

• To understand how the decision making process 
can be enhanced by modeling 

• To appreciate the practical benefits of 

management support systems in real business 
settings and in society 

• To obtain skills in spatial analysis 

• To become knowledgeable of the ArcView GIS 
software applications. Be able to implement 
hands-on basic GIS functions using ArcView 

• To appreciate how decision_makers in the public 
and private sectors can better assess and make 



experience and jobs, a more condensed curricular 
introduction was felt to be appropriate. The IS major 
is distinguished by two required courses that are not 
standard in IS '97: a course in geographic information 
systems (GIS) and a course in computer ethics. Both 
of these courses relate to strengths of the university: it 
is known for its strong GIS faculty and it has a liberal 
arts foundation that ethics relates to. 
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decisions on managerial problems through the use 
ofGIS 

• Understand the concepts and principles of 

structured and object oriented methodologies in 
systems development 

• Be able to apply effective system modeling tools 
and skills including CASE 

• Understand the perspective of complete life cycle 
activities, their value and application 

• Understand information systems from the 

perspective of pieces of an organizational system 

• Understand the magnitude of technological 

advances and their implications to ethical issues 

• Understand the theoretical perspective of ethical 
philosophies as they relate to business and 
organizations 

• Identify and understand examples of real world 
ethical dilemmas in information systems including 
the many sides of such dilemmas 

• Be able to discuss and properly communicate the 
basic concepts of information systems 

• Be able to design and build a business information 
system 

• Be able to develop and implement software in a 
business project solution 

• Communicate a complete systems project 
including verbally presenting and writing up and 
documenting the project 
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TABLE 2 

CURRICULUM OF BACHELOR OF SCIENCE 
IN INFORMATION SYSTEMS (BSIS) 



ISYS312 Productivity with Information 

Technology 

MGMT310 Philosophical Foundations of 
Management 

ISYS 219 Fundamentals of Structured 

Programming with C 

MGMT 330 Managing and Leading 

Organizations 

ISYS 317 Introduction to Programming 

Techniques: Object-Oriented 

Programming 

MTHW 303 Mathematical Applications 
for Information Systems 

BUAD 337 Political and Business Economics 

ISYS 318 Database Concepts 

BUAD 469 Accounting Information Systems 

ISYS 425 Management and Decision 
Systems 

ISYS 365 Geographic Information Systems: 

BUAD 340 Principles of Marketing 

ISYS 489 Applied Software Development 

Project II 

ISYS 415 Computer Ethics 

ISYS 404 Systems Analysis and Design 

ISYS 488 Applied Software Development 

Project I 

After the Redlands IS 97 curriculum had been run for 
two years, the IS faculty initiated a plan of program 
assessment that was carried out in early 2000. The 
assessment model consists of the following steps: (1) 
written survey of the 300 current students of the extent 
that their knowledge or competency of the program 
objectives and on their suggestions for program 
enhancements, (2) written survey of 700 alumni of the 
program on the extent that program objectives have 
helped them in their careers and on their suggestions 
for program enhancements, (3) focus group with the IS 
Degree Program Corporate Advisory Committee on 
their recommendations for curricular improvement, (4) 
focus groups with samples of current students about 
their suggestions for curricular improvements, and (5) 
IS faculty review and suggestions of current 
curriculum. 

The approach is multi-faceted, i.e., it includes many 
stakeholders and several methodologies. In particular, 



it gathers data from current students, alumni, corporate 
advisors, and IS faculty. It utilizes the methods of 
written survey, directed focus groups, and discussion. 
All of this information is aggregated together to form 
part of the IS program assessment report. Other 
information required by the university is added to the 
report including for instance data on the program, 
current status of the curricular and support services, 
advising, and math and writing skills. Altogether, this 
larger amount of data than is ordinarily collected 
provides a more in-depth insight into the program 
successes and failures, strengths and weaknesses, and 
feeds information into design of the future curriculum. 

Results of Assessment Procedures 

Program. The' BSIS program was assessed through 
three open-ended questions in two surveys, one of 
current students and a second of alumni. A third 
program assessment was conducted through two focus 
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groups with BSIS students. In the hour and a half 
focus group sessions, the students were first presented 
with the results of the BSIS curricular survey. Then 
there was a segment of discussion of the curricular 
results. Finally, there was a 45 minute open group 
discussion of suggestions for the BSIS program and 
curriculum. 

Only the survey responses responses from current 
BSIS students that made specific recommendations are 
included, i.e., a comment such as "great program" is 
not included because it does not inform the assessment 
process. Question 3 pertains especially to the 
program. There are several general points coming out 
of these comments regarding the program. On is that 
student would like to see the latest technology 
emphasized in the program. This implies that in 
program revision, attention needs to be given to the 
most current and important technical skills. Responses 
also support review for very recent and current 
textbooks underpinning the technical skills. There are 
several comments about workshop/tutorial preparation 
in math and writing skills as important. In general the 
administrative mechanics and support services of the 
program are not commented on, which would imply 
that they are working well. 

The summary of the two BSIS student focus groups 
underscore several points regarding the program. One 
is that the pre-entry counseling and advertising 
materials need to correctly position the program 
balance. In particular, the BSIS program offers a 
balanced blend of managerial and technical content. 
However, some incoming students have mis-read the 
program as strictly technical. Students are not clear on 
the level of technical expertise that they will be 
expected to achieve i.e., will they be intermediate 
programmers or master programmers? The program 
needs to articulate its goals early on. One comment in 
response is that the survey numerical responses 
indicate a considerable range of incoming technical 
knowledge/competency in particular between 1 and 
3.5 on a 1 to 5 Likert scale. Hence, part of the 
misunderstanding is that students technical mastery 
levels will remain different throughout the program 
and what can be standardized is the minimum level of 
technical knowledge/competency. 

Overall, the student suggstions/recommendations are 
centered on the technical content of the program; they 
are seeking expanded technical coverage. They appear 
satisfied with the program administration and 



academic support and service elements. 

Curriculum. The survey of current students 
measured Likert scale responses on the extent of 
current knowledge of 26 program objectives. For 
N~53 responses, the average current knowledge was 
2.59 on a 1 to 5 Likert scale, with 5 being the highest 
value. This reflects an intermediate state of current 
knowledge, which is to be expected for the full 
spectrum of students over the two years of the 
program. The comparison of students in the first half 
of the program i.e., junior year to those in the second 
half i.e., senior year shows an average current 
knowledge of juniors of 2.45 and of seniors of 2.73, 
for an average gain of 0.28, or eleven percent. When 
juniors are compared with students who were seniors 
in the last six months or had just completed the 
program, the average gain in current knowledge was 
0.57 or twenty three percent. These results provide 
confirmation of the learning value of the IS major. We 
interpret that the gains are not even higher because the 
student population consists of working adults students, 
most of whom come into the program as IS or 
technology professionals, so their starting knowledge 
base is higher than for traditional younger resident 
students. 

The gains in current knowledge from the program can 
be further disaggregated by the sixteen program 
objectives. Comparing juniors to seniors in the last 
nine months and a few very recent graduates, the 
following are the ranked gains by program objective. 





Current BSIS Students 




Rank 


BSIS Program Objective 


Av. Gain in 
Current 
Knowledge from 
Jr. Year to Last 9 
Mos. 


i 


Access in Development 


1.06 


2 


Dec. Making with GIS 


1.05 


3 


Communicate Project 


1.03 


4 


Database Applications 


0.98 


5 


Relational Database 


0.94 


6 


Methods of Syst Analysis 


0.94 


7 


CASE & Syst Tools 


0.92 


8 


Visual Basic 


0.79 


9 


Structured Programming 


0.78 


10 


Syst Life Cycle 


0.63 


11 


ArcView Software 


0.57 


12 


Internet & Web Applications 


0.56 
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13 


GUI Interface 


0.54 


14 


Communic. Basic Concepts 


0.53 


15 


C Language 


0.47 


16 


DSS/E1S 


0.46 


17 


Decision Making 


0.45 


18 


Design/Build an IS 


0.43 


19 


Spatial Analysis 


0.36 


20 


IS Project Solution 


0.35 


21 


Ethical Dilemmas 


0.33 


22 


Organiz. Systems 


0.31 


23 


Ethical Philosophies 


0.13 


24 


Ethic Implic of Techn. 


0.13 



and in the mainstay areas of data-bases and systems 
development. Also, CASE tools and visual basic are 
high. Those are areas in the present curriculum that 
appear to be working well and are likely to be retained 
in the new BSIS 2000 curriculum although 

modernized. 

On the other hand, areas that are not doing well as 
measured by the extent of knowledge gain are 
networking concepts, management support systems, 
certain aspects of computer ethics, organizational 
systems, and solutions with IS projects. These are 
pointing to areas in the BSIS 2000 curriculum in 
which to redesign courses and educational strategies 
and to seek major teaching and learning 
improvements. For instance, the low score for 
networking concepts may stem from the lack, until 
recently, of a hands-on networking laboratories for 
training on NT administration and configuration. 
Small practice networks have now been located in the 
Jones Center and all the major regional centers. Also, 
the networking course was in need of revision, a task 
which has recently been accomplished, but may need 
even more revision and attention in the new 

curriculum. 

There are a number of gender differences in 
knowledge/competency by program objectives. The 
statistically significant gender differences for 
competency/knowledge of current students are for: 
structured programming and C language (p=0.05) and 
there are lesser differences for networking and ethical 
dilemmas (p=0.10). In all these cases, the knowledge 
competency of males exceeds that of females. The 
programming and networking differences may relate 
to occupational differences prior to entry, while the 
difference in knowledge of ethical dilemmas may 
related to differential prior exposure to ethical 



25 


Mgt Support Systems 


0.10 


26 


Networking Concepts 


0.04 



The information in the disaggregated table is valuable 
in determining where the most learning is taking place 
and where learning is not as effective, so future 
improvements can be focused in those areas. The 
results confirm extensive learning in GIS decision 
making, a new area for most students, in 
communication of project, 



dilemmas in the real world. 

The knowledge/competency results can be grouped by 
major curricular areas. For instance, the Systems 
curricular area comprises the knowledge/competency 
objectives of methods of systems analysis, CASE and 
systems tools, systems life cycle, designing/building 
an information system, and IS project solution. When 
the knowledge/competency of current BSIS students is 
grouped by major curricular areas (see Figure 1), the 
highest levels of knowledge/competency are in 
communications/organization/ethics, while the lowest 
levels are in DDS/GIS and systems, with technical in 
the middle. This emphasizes that the students have 



3. SO 




range from 0 to 5, but actualy are in the range from 1 to 5. 
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considerable knowledge of the softer sides of the 
curriculum. Figure 1 also demonstrates a gender gap 
overall and for the four major knowledge/competency 
areas. Xtfeis mostly relate to the prior levels of 
experience an<~ 
entered the BSIS 

4.50 

have had more expc 
To a less 
differences in sfeflf 
Since t\£ gende 
differei§:e b£$r< 
regards tt^ reason 



by motivational factors i.e., that students are more 
motivated to learn about the technical and systems 
areas due for job and career related reasons. 
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FIGURE 2 

OVERALL KNOWLEDGE/COMPETENCY VS. 

TIME IN PROGRAM, CURRENT BSIS 
STUDENTS 
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Correlation analysis for the aggregated knowledge/ 
competency areas and with elapsed program duration 
in months shows signficant correlations among the 
overall knowledge competency and the four important 
areas (see Tables 3 and 4). This implies that generally 
a student is higher or lower in knowledge competency 
in all the areas i.e., they vary up and down together 
among individuals. This points to a more generic set 
of knowledge/competency across areas. This may 
reflect the real world experience of adult students. For 
instance, a student who had worked in a systems 
analysis position for ten years before entering the 
BSIS would have built up knowledge in all of the four 
major areas, while an entering student without any IS 
job experience would be weak in the technical, 
systems, and DSS/GIS areas. It is somewhat less clear 
why he/she would be weak in 
communications/organizations/ethics as well, but that 
is the case. 

The correlations also show the correlation with elapsed 
time in the program. There is a statistically significant 
correlations of overall and technical knowledge/ 



competency with elapsed time, but not with the other 
areas. This is seen in the scatter plot shown in Figure 
2. What is clear in all these areas is that there is a 
large range of incoming competencies; again this 
relates to the population of adult students. For overall 
knowledge (Fig. 1) and technical knowledge, there is a 
significant trend of increase over the two years. Yet, 
at graduation, there is still substantial range of 
knowledge/competency. The trend line has moved up, 
but the high variation remains. This is helpful in 
explaining teaching challenges even late in the BSIS 
program of a large range of competencies in a single 
cluster group. We interpret this as continuation of 
incoming ranges. By contrast, for 

communications/organizations/ethics, the average 
incoming knowledge/competency is higher but with an 
even larger range and doesn’t increase much with the 
same large range throughout the program. 

The data from the alumni survey were equally 
revealing. The key question related to program 
objectives is different; it asked how well the IS degree 
had impacted the alumni's career advancement. The 
following were the ranked program objectives (on a 1 
to 5 Likert scale, with 5 being the highest impact): 

The alumni results point to the importance of the 
"softer" sides of the curriculum on career impact — 
areas such as communications, organizations, 
designing and building an IS project, and the "building 
block" areas of systems development and database. 
The top two objectives are the area of interpersonal 
communications. This result corresponds to other 
studies in the IS educational literature that stress the IS 
career advances of 



200 Proceedings of the 1 5 th Annual Conference of the International Academy for Information Management 




9 



interpersonal skills and teamwork (see literature 
review in Pick and Schenk, 1993). In general, this 
literature points to the "softer" areas as ultimately 
having more impact on IS graduates careers in the long 
term. It is important to ask why these behavioral skills 
rise to such importance. One reason may be the rising 

TABLE 3 

CORRELATIONS OF IMPORTANT 
KNOWLEDGE/COMPETENCY AREAS FOR 
CURRENT BSIS 



career attainment of BSIS alumni, so that eleven 
percent of the alumni are in top management 
(President, VPs) and forty percent are in middle 
management or higher. In middle to top management 
jobs, the areas of communications, teamwork, and 
organizations are essential. 



Months Overall Technical DSS/GIS Systems C/O/E 



Months 


1 








Overall 


0.309* 


1 






Technical 


0.349** 


0.923** 


1 




DSS/GIS 


0.273 


0.863** 


0709** 


1 


Systems 


0.251 


0.923** 


0.798** 


0.732** 1 


C/O/E 


0.186 


0.891* 


0.682** 


0.771** 0.852** 



* significant at' 0.05 level 
** significant at 0.01 level 

LE 4 

CORRELATIONS OF CAREER IMPACT ON 
ALUMNI FOR IMPORTANT BSIS 
CURRICULAR AREAS 



Year Overall Technical DSS/GIS Systems C/O/E 

Graduated 

Year 1 



Graduated 



Overall 


-0.187 


1 






Technical 


-0.047 


0.818** 


1 




DSS/GIS 


-0.287* 


0.754** 


0.436** 


1 


Systems 


-0.235 


0.817** 


0.579** 


0.519** 1 


C/O/E 


-0.202 


0.746** 


0.351** 


0.560** 0.623** 



1 



* significant at 0.05 level 
** significant at 0.01 level 





Alumni 






BSIS Program Objective 


Impact of BSIS 
Program 
Objective on 

Alumni's Career 


Rank 




Advancement 



1 


Communicate Project 


4.08 


2 


Communic. Basic Concepts 


4.04 


3 


Syst Life Cycle 


4.00 


4 


Organiz. Systems 


3.98 


5 


IS Project Solution 


3.97 
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6 


Design/Build an IS 


3.92 


7 


Relational Database 


3.84 


8 


Database Applications 


3.79 


9 


Ethical Dilemmas 


3.75 


10 


Ethic Implic of Techn. 


3.73 


11 


Ethical Philosophies 


3.69 


12 


Access in Development 


3.63 


13 


Networking Concepts 


3.53 


14 


Mgt Support Systems 


3.50 


15 


Methods of Syst Analysis 


3.36 


16 


Decision Making 


3.31 


17 


Dec. Making with GIS 


3.23 


18 


CASE & Syst Tools 


3.22 


19 


GUI Interface 


3.13 


20 


Visual Basic 


3.12 


21 


Spatial Analysis 


3.03 


22 


DSS/EIS 


3.01 


23 


Structured Programming 


2.98 


24 


Internet & Web Applications 


2.88 


25 


ArcView Software 


2.13 


26 


C Language 


1.95 



Systems life cycle is in third place, fifth, and sixth 
place interspersed between interpersonal 
communication and organizational knowledge. Data- 
base design and concepts are sixth and seventh. From 
the standpoint of curricular revision, this points to the 
need to put even greater emphasis on these behavioral, 
organizational, and ethical areas. These are long time 
mainstays of the systems field and model curricula. 
Systems develop-ment has proven to be a robust 
knowledge area over a number of iterations of national 
model curricula (DPMA, 1990, 1997). It is likely that 
systems development will be retained in the BS1S 
2000 curriculum, although it will take a different form 
i.e., it will likely be more oriented towards rapid 
development in the context of the web and 
dynamically changing environments. 

Database is the most valued technical curricular area 
for alumni career advancement. This again reflects an 
enduring mainstay feature of national model curricula. 
Databases have persisted in importance in IS. They 
remain so in the internet age, since database underlies 
many aspects of e-commerce and e-business. 



Alumni rate the career impact of ethics in the upper 
moderate range. It may be that BSIS graduates, with 
rising career attainment and often managerial or high 
level systems status, run into ethical issues to an 
increasing extent. 

At the lower end of perceived career benefits are a 
number of technical areas including C language, 
ArcView software, internet and web applications, and 
structured programming. By contrast, those technical 
areas were all in the middle range for current student's 
curricular knowledge gain. There are several 
explanations. First, some of these areas are so new 
(ArcView, internet) that most of the graduates could 
not have been exposed to them in the BSIS or outside 
of the university at the time of their BSIS schooling. 
The C language and structured programming may be 
less important because of the occupational rise of 
alumni towards management and senior systems 
positions that involve less hands-on programming. 
This is reflected in the respondent profiles which show 
that only eight percent of alumni are in programmer 
positions. At the same time, only two percent of the 
sample of current students are programmers. 

The lower perceived career value of technical 
curricular objectives underscores the fast movement of 
technology, so future BSIS alumni may not be able to 
utilize the technical skills they are learning today on a 
career basis. This is contrary to the sometimes 
strongly stated desire of current students to master 
technical skills. 

Generally, the alumni findings on specific curricular 
areas point to the importance of the "softer" behavioral 
elements in the curriculum as well as the long-time 
mainstays of systems development and database. 

The results by gender indicate that alumni male BSIS 
students have greater career impact of the program. 
The only statistically significant difference is for the 
career impact of communicating the project, with 
males having higher impact. This is unexplained. 

The results of the first open-ended question for current 
BSIS students provide valuable additional 
suggestions/comments on the curriculum. In 
particular, question 1 concerns improvements in the 
BSIS curriculum. The suggestions focus on (1) 
expanding and improving the technical content, (2) 
adding optional workshops, tutorials, and elective 
courses that might even include certificate training, 
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such as Microsoft SCCE. In the technical realm, 
students are particularly interested in expanding the 
Visual Basic coverage, perhaps by adding another 
course. 

The results of the second open-ended question concern 
the value of the non-major courses in the BSIS 
program. There seems to be diverse opinions about 
Mgmt. 310 (Philosophical Foundations of 
Management) ranging from strong support to 
dislike/request to eliminate it. Looking at the parts of 
Mgmt 310, the humanities segment seemed more 
supported than the portfolio part. There was also a 
suggestion to separate the course into two courses and 
move the portfolio part later in the curriculum. The 
student response on the Mthw 303 (Mathematical 
Applications for Information Systems) course was 
mixed. We can assume that this may depend on the 
extent of individual student preparation and particular 
instructors. Reflecting other responses, some students 
may need tutorials and workshops to enhance 
readiness to take Mthw 303. The responses on the 
Mgmt. 330, Buad 337, and Buad 469 courses were 
generally good. 

Another aspect of the assessment of student work in 
the BSIS Program is the student experience with 
senior project. An evaluation was conduced by Jeff 
Kim of a random sample consisting of BSIS senior 
projects, with 3 project reports in 1998; 3 in 1999; and 
4 projects from 2000. The findings indicated the 
following accomplishments in the senior project: (1) 
high level of systems implementation. (2) 
documentation report was visible, and (3) students 
followed the project report structure offered by the 
project guidelines. However, the evaluation pointed to 
the following areas of weakness or challenge. No 
analysis was done on the real world organizational 
context of the projects. This is important for several 
reasons. Studies are increasingly pointing to the 
organizational aspect as important to systems success. 
Second, the organizational context touches based as a 
capstone element with the organizational/ managerial 
side of the BSIS curriculum. The BSIS 3 year goals in 
section 9 point to this area. Secon, no analysis was 
done in the sample projects on workflow, except in the 
case of “Use Case” analysis. Again, this is an 
important concept today for the practical success of 
systems, and needs to be emphasized in project 
enhancements in the future. Third, each stage of the 
project design and implementation appeared to be 
disjointed. In other words, the project calendar of 



“piece work” assignments that are put together at the 
end may be discouraging an integrated series of 
project tasks that are coordinated together. This can 
be improved by modifying the BSIS project guide to 
stress returning sometimes to earlier concepts and 
prior activities and steps, and to generally being more 
holistic. Lastly, the project reports did not reflect 
enough of what the students had learned from the 
exercise of doing a project, with the exception of one 
report written in 2000. This is a part of “reflection,” 
which correlates with missions and would add to the 
learning and experience of projects. 

The goals and objectives later point to areas for 
improvement of the senior project. The project 

assessment also points to the need to assess the timing 
and sequence of the project and project related (e.g. 
ISYS 404) courses; this can be considered with the 
present curriculum, prior to the BSIS 2000 

implementation. A final point is that the focus of 
knowledge and learning may need to be refocused. 
We need to think through the goal overall of the 
project. Is it to perform systems analysis, write up 
requirements, program, and evaluate the outcome? Is 
it keyed to preparation for jobs and hiring on 
graduation? Or is it to reflect on the principles of 
systems design? These need to be thought through 
more and articulated to students better, as well as 
practiced in the way decided upon. 

As an example of the assessment findings, the survey 
of current students received Likert scale responses on 
the extent of current knowledge of 26 program 
objectives. For N=53 responses, the average current 
knowledge was 2.66 on a 1 to 5 Likert scale, with 5 
being the highest value. This reflects an intermediate 
state of current knowledge, which is to be expected for 
an even distribution of students over the two years of 
the program. The comparison of students in the first 
half of the program i.e., junior year to those in the 
second half i.e., senior year shows an average current 
knowledge of juniors of 2.55 and of seniors of 2.79, 
for an average gain of 0.25, or about ten percent. 
When juniors are compared with students who were 
seniors in the last six months or had just completed the 
program, the average gain in current knowledge was 
0.42 or about sixteen percent. These results provide 
confirmation of the learning value of the IS major. We 
interpret that the gains are not higher because the 
student population consists of working adults students, 
most of whom come into the program as IS or 
technology professionals, so their starting knowledge 
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base is higher than for a traditional younger resident 


i 


GIS 


1.36 


student. 


2 


Communication of a 
major project 


1.12 


The gains in current knowledge from the program can 


3 


Relational Data-base 


1.12 


be further desegregated by the sixteen program 


4 


Systems Dev. Life Cycle 


0.95 


objectives. Comparing juniors to seniors in the last six 


5 


CASE tools 


0.94 


months or recent graduates, the following are the 


6 


Data-base Applications 


0.92 


highest gains by program objective. 


7 


Verbal Communications 


0.58 



Doing the same comparison, the areas of lowest gains 
(or losses) were the following. 

The information in the disaggregated table is 
invaluable in determining where the most learning is 
taking place and where learning is not as effective, so 
future improvements can be focused in those areas. 
For instance, the results confirm extensive learning in 
GIS, a new technology area for most students, in 
verbal communications, and in the mainstay areas of 
data-bases and systems development. Those are areas 
in the present curriculum that appear to be working 
well and will likely be retained in the new curriculum 
although modernized. 

On the other hand, areas that are not doing well as 
measured by where learning is taking place are the 
world wide web, networks, organizational systems, 
certain aspects of computer ethics, and DSS/EIS. 
These are pointing to areas in the new curriculum to 
seek major teaching and learning improvements. For 
instance, the university has been behind schedule on 
implementing its high speed campus network, which 
until recently has reduced the potential for learning of 
world wide web skills. Given the importance of this 
area in industry, it is something that is urgently 
pointed to for the new curriculum. The low score for 
networking may stem from the lack, until recently, of a 
hands-on networking laboratory for training on NT 
administration and configuration. Also, the 
networking course was in need of revision, a task 
which has recently been accomplished, but may need 
even more revision and attention in the new 
curriculum. 



Rank 


Program Objective 


Average gain in 
current 

knowledge from 
junior year to 
last 6 mos of 






curriculum plus 
recent grads 



Rank 


Program Objective 


Average gain in 
current knowledge 

from junior year to 
last 6 mos of 
curriculum plus 

recent grads 


21 


Dec Supp Systems/ 
Exec Inf. Systems 


0.13 


22 


Understanding the 
Implications of 

technology 
advances for ethical 
issues 


0.12 


23 


Understanding the 
theoretical 
perspective of 

ethical philosophies 
as they relate to 
businesses and 

organizations 


0.10 


24 


Organizational 

Systems 


-0.12 


25 


Networks 


-0.24 


26 


World Wide Web 


-0.62 



The data from the alumni survey were equally 
revealing. For instance, on the question of how well 
the IS degree had impacted the alumni’s career 
advancement, the following were the ten highest 
program objectives (on a 1 to 5 Likert scale, with 5 
being the highest impact). 



Rank 


Program Objective 


Rating 


of 






Impact 


of 






Program 

Objective 

Alumni's 

Career 


on 






Advancement 


i 


Verbal Communications 


4.04 
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Communication of a 
Major Project 


4.04 


3 


Organizational Systems 


3.96 


4 


Systems of Life Cycle 


3.96 


5 


Senior Project 


3.93 


6 


Relational Data-Base 


3.89 


7 


Be Able to Design and 
Build a Business 

Information System 


3.88 


8 


Data-Base Applications 


3.84 


9 


Understanding the 

Implications of 

Technology Advances for 
Ethical Issues 


3.72 


10 


Understanding Ethical 

Delemmas 


3.72 



The alumni results point to the importance of the softer 
sides of the curriculum on career impact — areas such 
as communications, organizations, conducting a 
project, and ethical issues. Systems life cycle is in 
between in ’’softness" and is ranked a high fourth. The 
more technical areas of data-base are present but 
ranked sixth and eighth. This confirms some other 
literature that have pointed to the "softer" areas as 
ultimately having more impact on IS graduates 
careers. From the standpoint of curricular revision, 
this points to the need to put even greater emphasis on 
these behavioral, organizational, and ethical areas. 

The results from the focus group sessions with 
students and the discussions with the corporate 
advisory committee were equally informative, but are 
not covered in this extended abstract, but will be in the 
full paper. Suffice it to say that they delved into 
certain issues in even more depth and provide even 
greater knowledge of current status of curriculum and 
the recommendations for the new one. 

In summary, this paper presents a new model for IS 
program assessment, with an emphasis on curriculum. 
Many stakeholders can be involved in a more 
extensive data-gathering process that provides more 
robust information on student learning, alumni career 
benefits from their education, and corporate advice for 
program and curricular improvement and change. The 



assessment process serves to set a baseline than can be 
re-tested longitudinally over time to benchmark longer 
term program advances. The methods can be adapted 
for a wide variety of IS undergraduate and graduate 
programs. In another paper we suggest more detailed 
steps that programs can take to instill a full program 
and curricular assessment process into their strategic 
planning and development. 
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